Multiplicative stochasticity such as Dropout improves the robustness and gener-alizability deep neural networks. Here, we further demonstrate that always-on multiplicative stochasticity combined with simple threshold neurons provide a sufficient substrate for deep learning machines. We call such models Neural Sampling Machines (NSM). We find that the probability of activation of the NSM exhibits a self-normalizing property that mirrors Weight Normalization, a previously studied mechanism that fulfills many of the features of Batch Normalization in an online fashion. The normalization of activities during training speeds up convergence by preventing internal covariate shift caused by changes in the distribution of inputs. The always-on stochasticity of the NSM confers the following advantages: the network is identical in the inference and learning phases, making the NSM a suitable substrate for continual learning, it can exploit stochasticity inherent to a physical substrate such as analog non-volatile memories for in memory computing, and it is suitable for Monte Carlo sampling, while requiring almost exclusively addition and comparison operations. We demonstrate NSMs on standard classification benchmarks (MNIST and CIFAR) and event-based classification benchmarks (N-MNIST and DVS Gestures). Our results show that NSMs perform comparably or better than conventional artificial neural networks with the same architecture.

Related work is well cited across these fields, and the approach is unique in this reviewer's knowledge. Quality The quality of this work is adequate, though there are a couple of simple errors in the text (misspelling in Figure 1, missing sections in the supplementary material, lack of explanation of some abbreviations such as W_3 and S2M). Overall, the text and derivation is done with high quality, and the tricks used in the derivation are called out to adequately describe the steps to the reader. The conclusions stand on their own, and quality of insight is needed to bridge stochastic neural networks, multiplicative weights, and weight normalization. The work could use more difficult datasets, though, to emphasize these results.

The author rebuttal went a long way towards satisfying the reviewers and all of them have recommended acceptance after discussions. The authors should go through the suggestions given by the reviewers (esp.

Multiplicative stochasticity such as Dropout improves the robustness and gener- alizability deep neural networks. Here, we further demonstrate that always-on multiplicative stochasticity combined with simple threshold neurons provide a suf- ficient substrate for deep learning machines. We call such models Neural Sampling Machines (NSM). We find that the probability of activation of the NSM exhibits a self-normalizing property that mirrors Weight Normalization, a previously studied mechanism that fulfills many of the features of Batch Normalization in an online fashion. The normalization of activities during training speeds up convergence by preventing internal covariate shift caused by changes in the distribution of inputs.